Dimensions to Analyze Applications

Applications should be an invaluable experimental source of information and challenges for AI research. The real world always stretches the limitations of our AI systems, pointing toward new research themes in many areas. In the process of implementing an application, the designer continually makes choices based on (1) the baseline architecture used to implement the application, (2) the characteristics of the problem itself, or (3) arbitrary decisions and assumptions. All these decisions are intertwined in the resulting application, and as a result, it is not easy to abstract a description of the functionality provided by the architecture itself. At the same time, we w ould like to base our science on real-world applications that are subject to controllable experiments whose parameters can be modied to obtain experimental results of our programs' behavior. However, real applications rarely facilitate this task. Although the AI community has developed formalisms to describe AI architectures, we are still lacking a formal language to describe tasks and problems that provides a good qualitative understanding of AI applications. We argue that this is a major deciency that stops feedback from applications to research. This work is an eort towards descriptions of applications in terms that are useful 1) to extract conclusions from particular implementations , 2) to facilitate cross-comparisons among dierent architectures applied to the same problem, and 3) to facilitate comparisons among dierent tasks. We analyze the Sisyphus experience, and we propose a set of dimensions to describe applications that distinghish between descriptions of the properties of the architecture, the type of problem, and the data sets. We show h o w these dimensions can be used to produce useful distinctions in the context of the rst Sisyphus task, an oce assignment problem. Our hope is that the same dimensions will be useful to other researchers in describing, characterizing, and producing qualitative e v aluations of their applications, as well as a useful point of comparison for future Sisyphus eorts.